Multi-cluster Ingress

ABSTRACT

A method for load balancing application requests across a multi-cluster container orchestration system includes receiving a load-balancing configuration for a multi-cluster service managing access to a set of destination clusters hosting a software application deployed by a user. The multi-cluster service uses the load-balancing configuration to load balance application level traffic across the set of destination clusters. Each destination cluster includes at least one container executing the software application and a respective geographical region. The method also includes receiving an application level request directed toward the software application hosted across the destination clusters. The application level request is received from a client and includes a host name and a geographical location associated with the client. The method also includes routing the request to one of the clusters based on the geographical location of the request and the respective geographical locations of the clusters.

CROSS REFERENCE TO RELATED APPLICATIONS

This U.S. patent application is a continuation of, and claims priorityunder 35 U.S.C. § 120 from, U.S. patent application Ser. No. 16/372,220,filed on Apr. 1, 2019. The disclosure of this prior application isconsidered part of the disclosure of this application and is herebyincorporated by reference in its entirety.

TECHNICAL FIELD

This disclosure relates to a multi-cluster ingress for a containerizedorchestration system.

BACKGROUND

Some cloud-based services (via distributed systems) offer containerizedorchestration systems. These systems have reshaped the way software isdeveloped, deployed, and maintained by providing virtual machine-likeisolation capabilities with low overhead and high scalability. Softwareapplications execute in secure execution environments (e.g., containersor pods) and co-located pods may be grouped into clusters, each clusterisolated from other clusters. Load Balancers are commonly used toimprove the distribution of traffic and workloads across pods within acluster. Layer 7 (L7) load balancing, (i.e., application layer) loadbalances the actual content of messages. For example, an L7 loadbalancer might operate on HyperText Transfer Protocol (HTTP) orHyperText Transfer Protocol Secure (HTTPS), and make routing decisionson the content of the message. Load balancers for containerizedorchestration systems are typically L7 load balancers that operate on asingle cluster.

SUMMARY

One aspect of the disclosure provides a method for load balancingapplication requests across a multi-cluster containerized orchestrationsystem. The method includes receiving, at data processing hardware, aload-balancing configuration for a multi-cluster service managing accessto a set of destination clusters hosting a software application deployedby a user. The multi-cluster service is configured to use theload-balancing configuration to load balance application level trafficassociated with the software application across the set of destinationclusters. Each destination cluster includes at least one containerexecuting the software application and a respective geographical regionthat is the same or different than at least one other geographicalregion associated with another one of the destination clusters in theset of destination clusters. The method also includes receiving, at thedata processing hardware, an application level request directed towardthe software application hosted across the set of destination clusters.The application level request is received from a client and includes ahost name and a geographical location associated with the client. Themethod also includes routing, by the data processing hardware, theapplication level request to one of the destination clusters in the setof destination clusters based on the geographical location of theapplication level request and the respective geographical regions of theset of destination clusters.

Implementations of the disclosure may include one or more of thefollowing optional features. In some implementations, routing theapplication level request includes determining which destination clusterin the set of destination clusters is closest to the geographicallocation associated with the client of the application level requestbased on the respective geographical regions of the set of destinationclusters and routing the application level request to the destinationcluster in the set of destination clusters having the respectivegeographical region that is closest to the geographical locationassociated with the client of the application level request. In someexamples, routing the application level request is further based onrespective load balancing attributes specified by the multi-clusterservice for each destination cluster in the set of destination clusters.The received load-balancing configuration may include a user-derivedservice name that uniquely identifies the multi-cluster service.

In some implementations, the method includes identifying, by the dataprocessing hardware, cluster selection criteria specified by themulti-cluster service for choosing clusters from a cluster registry thatwill serve application level traffic for the multi-cluster service andselecting, by the data processing hardware, the set of destinationclusters from the cluster registry based on each destination cluster inthe set of destination clusters having a respective set of one or morelabels that satisfy the cluster selection criteria specified by themulti-cluster service. The cluster selection criteria specified by themulti-cluster service may include at least one of one or moreequality-based matching requirements or one or more set-based matchingrequirements. Optionally, the method further includes, for eachdestination cluster in the set of destination clusters, instantiating,by the data processing hardware, a corresponding derived service withinthe destination cluster. The derived service is configured to create acorresponding network endpoint group (NEG) that includes a group ofendpoints. Each endpoint in the group of endpoints is associated with arespective container of the destination cluster and includes arespective internet protocol (IP) address and a respective port fordistributing application level traffic directly to the respectivecontainer.

Each corresponding derived service includes, in some implementations, aunique derived service name that is different than the derived servicenames of the other derived services. The derived service name has atrimmed service name portion and a unique hash portion. The trimmedservice name portion includes a user-derived service name of themulti-cluster service and the unique hash portion includes a unique hashof the user-derived service name of the multi-cluster service. Themethod, in some examples, further includes, in response to receiving theapplication level request, accessing, by the data processing hardware, auniform resource locator (URL) mapping. The URL mapping specifies a listof one or more host names that map to a service of the one or moredestination clusters. The method also includes determining, by the dataprocessing hardware, whether the host name of the received applicationlevel request includes one of the host names in the list of one or morehost names specified by the URL mapping, and when the host name of thereceived application level request includes one of the host names in thelist, forwarding, by the data processing hardware, the receivedapplication level request to the service.

The application level traffic may include HyperText Transfer Protocol(HTTP). The application level traffic may also include HyperTextTransfer Protocol Secure (HTTPS) protocol. At least a portion of theapplication level request optionally includes a transport layer security(TLS) protocol. The method, in some implementations, further includes,prior to routing the application level request and for each destinationcluster in the set of destination clusters, determining, by the dataprocessing hardware, whether a number of application level requestscurrently routed to the destination cluster satisfies a maximum requestrate, and when the number of application level requests satisfies themaximum request rate, preventing routing of the application levelrequest to the destination cluster.

Another aspect of the disclosure provides a system for load balancingapplication requests across a multi-cluster containerized orchestrationsystem. The system includes data processing hardware and memory hardwarein communication with the data processing hardware. The memory hardwarestores instructions that when executed on the data processing hardwarecause the data processing hardware to perform operations. The operationsinclude receiving a load-balancing configuration for a multi-clusterservice managing access to a set of destination clusters hosting asoftware application deployed by a user. The multi-cluster service isconfigured to use the load-balancing configuration to load balanceapplication level traffic associated with the software applicationacross the set of destination clusters. Each destination clusterincludes at least one container executing the software application and arespective geographical region that is the same or different than atleast one other geographical region associated with another one of thedestination clusters in the set of destination clusters. The operationsalso include receiving an application level request directed toward thesoftware application hosted across the set of destination clusters. Theapplication level request is received from a client and includes a hostname and a geographical location associated with the client. Theoperations also include routing the application level request to one ofthe destination clusters in the set of destination clusters based on thegeographical location of the application level request and therespective geographical regions of the set of destination clusters.

This aspect may include one or more of the following optional features.In some implementations, routing the application level request includesdetermining which destination cluster in the set of destination clustersis closest to the geographical location associated with the client ofthe application level request based on the respective geographicalregions of the set of destination clusters and routing the applicationlevel request to the destination cluster in the set of destinationclusters having the respective geographical region that is closest tothe geographical location associated with the client of the applicationlevel request. In some examples, routing the application level requestis further based on respective load balancing attributes specified bythe multi-cluster service for each destination cluster in the set ofdestination clusters. The received load-balancing configuration mayinclude a user-derived service name that uniquely identifies themulti-cluster service.

In some implementations, the operations include identifying clusterselection criteria specified by the multi-cluster service for choosingclusters from a cluster registry that will serve application leveltraffic for the multi-cluster service and selecting the set ofdestination clusters from the cluster registry based on each destinationcluster in the set of destination clusters having a respective set ofone or more labels that satisfy the cluster selection criteria specifiedby the multi-cluster service. The cluster selection criteria specifiedby the multi-cluster service may include at least one of one or moreequality-based matching requirements or one or more set-based matchingrequirements. Optionally, the operations further include, for eachdestination cluster in the set of destination clusters, instantiating acorresponding derived service within the destination cluster. Thederived service is configured to create a corresponding network endpointgroup (NEG) that includes a group of endpoints. Each endpoint in thegroup of endpoints is associated with a respective container of thedestination cluster and includes a respective internet protocol (IP)address and a respective port for distributing application level trafficdirectly to the respective container.

Each corresponding derived service includes, in some implementations, aunique derived service name that is different than the derived servicenames of the other derived services. The derived service name has atrimmed service name portion and a unique hash portion. The trimmedservice name portion includes a user-derived service name of themulti-cluster service and the unique hash portion includes a unique hashof the user-derived service name of the multi-cluster service. Theoperations, in some examples, further include, in response to receivingthe application level request, accessing a uniform resource locator(URL) mapping. The URL mapping specifies a list of one or more hostnames that map to a service of the one or more destination clusters. Theoperations also include determining whether the host name of thereceived application level request includes one of the host names in thelist of one or more host names specified by the URL mapping, and whenthe host name of the received application level request includes one ofthe host names in the list, forwarding the received application levelrequest to the service.

The application level traffic may include HyperText Transfer Protocol(HTTP). The application level traffic may also include HyperTextTransfer Protocol Secure (HTTPS) protocol. At least a portion of theapplication level request optionally includes a transport layer security(TLS) protocol. The operations, in some implementations, furtherinclude, prior to routing the application level request and for eachdestination cluster in the set of destination clusters, determiningwhether a number of application level requests currently routed to thedestination cluster satisfies a maximum request rate, and when thenumber of application level requests satisfies the maximum request rate,preventing routing of the application level request to the destinationcluster.

The details of one or more implementations of the disclosure are setforth in the accompanying drawings and the description below. Otheraspects, features, and advantages will be apparent from the descriptionand drawings, and from the claims.

DESCRIPTION OF DRAWINGS

FIG. 1 is a schematic view of an example system for load balancingapplication level traffic across multiple clusters of a containerizedorchestration system.

FIG. 2 is a schematic view of an example multi-cluster controller of thesystem of FIG. 1.

FIGS. 3A and 3B are schematic views of exemplary components of acontainer load balancer including network endpoint groups.

FIG. 4 is a schematic view of an example multi-cluster ingress of thesystem of FIG. 1.

FIG. 5 is a flowchart of an example method for conserving resources incontainerized systems.

FIG. 6 is a schematic view of an example computing device that may beused to implement the systems and methods described herein.

Like reference symbols in the various drawings indicate like elements.

DETAILED DESCRIPTION

Containerized applications, and the systems that orchestratecontainerized applications, are becoming increasingly popular due to, atleast in part, advances in remote and distributed computing.Containerized applications (i.e., virtualization) allow for theexistence of isolated user or application space instances. Each instance(i.e., container) may appear to the application as its own personalcomputer with access to all the resources necessary to execute (e.g.,storage, network access, etc.). However, applications in containers willonly be able to view and access resources assigned to its respectivecontainer. This facilitates security, mobility, scaling, and upgradingof applications in a distributed or cloud environment.

A container typically will be limited to a single application or processor service. Some container-orchestration systems deploy pods as thesmallest available computing unit. A pod is a group of one or morecontainers, each container within the pod sharing isolation boundaries(e.g., IP address). Controllers control resources in pods. Controllersare responsible for monitoring the health of pods, containers, andresources (and recreating the pods/containers if necessary). Controllersare also responsible replicating and scaling pods, as well as monitoringfor external (to the pod) events.

Because pods are typically temporary and fungible resources, they arefrequently created and destroyed (i.e., scaled in or out). Because somepods (i.e., backends) provide functionality to other pods (i.e.,frontends), services are created to allow frontends to track whichbackends provide the required functionality for the frontend. A serviceis an abstraction that defines a logical set of pods and a policy bywhich to access them. That is, one or more pods is targeted by a servicethat tie the backends to a corresponding frontend. The service maytarget pods that match a selection criteria. In some examples, theselection criteria includes label selection. That is, pods may includelabels and the service may select the desired pods by equality-based orset-based label matching.

A single physical machine (i.e., computer or server) hosts one or morecontainers (e.g., pods). The container-orchestration system will oftencoordinate multiple containerized applications across a many pods usinga cluster of physical machines. Typically, each machine in the clusteris co-located (i.e., the machines are geographically located near eachother) with one or more machines functioning as a master server and theremaining machines functioning as nodes. The master server acts as theprimary control plane and gateway for the cluster by, for example,exposing an Application Programming Interface (API) for clients, healthchecking the nodes, orchestrating communication, scheduling, etc. Thenodes are responsible for accepting and executing workloads using localand external resources and each node creates and destroys containers asinstructed by the master server. Clients interact with the cluster bycommunicating with the master server (e.g., directly or via libraries).The nodes within the cluster are generally isolated and segregated fromcontact outside of the cluster except as allowed by the master server.

Load balancing improves the distribution of workloads across multiplecomputing resources, and due to the distributed nature ofcontainer-orchestration systems, distributed systems frequentlyimplement Layer 7 (L7) load balancing. Layer 7 load balancing operatesat the high-level application layer (i.e., Layer 7), which involves theactual content of transmitted messages. HyperText Transfer Protocol(HTTP) and HyperText Transfer Protocol Secure (HTTPS) are thepredominant L7 protocols for website traffic on the Internet. Because ofthe high level, L7 load balancers may route network traffic in a moresophisticated way than other layer load balancer (e.g., Layer 4 loadbalancers). Generally, a L7 load balancer terminates the network trafficand analyzes the message content within the traffic. The L7 loadbalancer may then route the traffic based on the content of the message(e.g., based on an HTTP cookie). The L7 balancer may then create a newconnection to the proper destination node.

Current container-orchestration systems typically only offer L7 loadbalancing that target a single cluster. That is, each cluster requires aseparate load balancer that requires individual configuration andtraffic can only be balanced within the single cluster. To route trafficto an appropriate cluster (e.g., the cluster nearest geographically tothe source client), separate domains may be needed. For example,asia.shopping.com may route to a cluster located in Asia whileeurope.shopping.com may route to a cluster in Europe. Thus, it would beadvantageous for a load balancer that may serve highly-available,globally-distributed L7 services across multiple clusters in acontainer-orchestration system. Continuing with the example, the loadbalancer servicing multiple clusters could route a HTTP(S) request forshopping.com to either the cluster in Asia or the cluster in Europebased on a source of the HTTP(S) request and/or capacity at theclusters.

Implementations herein are directed toward a multi-cluster load balancerof a container-orchestration system to load balance application leveltraffic associated with a software application across a set ofdestination clusters. The multi-cluster load balancer receives aload-balancing configuration for a multi-cluster service that managesaccess to the set of destination clusters. As used herein, theload-balancing configuration may be referred to as an ingressconfiguration. Each destination cluster includes at least one podexecuting the software application in a secure execution environment(i.e., at least partially isolated from other pods or clusters) and arespective geographical region. In some scenarios, the at least onepod/container executes the software application in a non-secureenvironment. Each cluster may have a different geographic region. Themulti-cluster load balancer receives an application level requestdirected toward the software application hosted across the set ofdestination clusters and the load balancer routes the application levelrequest to one of the destination clusters based on the geographicallocation of the application level request and the respectivegeographical regions of the set of destination clusters. Thus, the loadbalancer targets multiple clusters while providing a single point ofmanagement and configuration across all of the clusters. The loadbalancer may take advantage of container-native load balancing (i.e.,distributing traffic directly to pods) and provides high availabilityfor hosted services if a cluster goes offline.

Referring now to FIG. 1, in some implementations, an example system 100includes a remote system 114. The remote system 114 may be a singlecomputer, multiple computers, or a distributed system (e.g., a cloudenvironment) having scalable/elastic computing resources 118 (e.g., dataprocessing hardware) and/or storage resources 116 (e.g., memoryhardware). The remote system 114 communicates with, via network 112 a,one or more clusters 120, 120 a-n, and each cluster 120 includes one ormore pods 122, 122 a-n, each executing one or more applications 124.While the examples herein describe the clusters 120 including one ormore pods 122, the clusters 120 may include any type of containers forexecuting the one or more software applications 124 without departingfrom the scope of the present disclosure. In some examples, part or allof one or more of the clusters 120 executes on the remote system 114.Some pods 122 may execute the same applications 124, while some pods122, within the same cluster 120 or a different cluster 120, may executedifferent applications 124. For example, each cluster 120 may includepods 122 that execute a shopping application 124. A service 123represents one or more applications 124 executing on multiple pods 122within the same cluster 120. To continue the previous example, ashopping service 123 may use the shopping application 124 that isexecuting on multiple pods 122. For example, all pods 122 executing theshopping application 124 may be associated with the shopping service123, and each respective pod 122 may be a fungible resource to fulfill arequest 30 to use the shopping service 123.

Each cluster 120 is also associated with a respective geographicalregion 121, 121 a-n. For example, the cluster 120 a may be associatedwith the geographical region 121 a of Asia, the cluster 120 b may beassociated with the geographical region 121 b of Europe, and the cluster120 n may be associated with the geographical region 121 n of NorthAmerica. That is, each cluster 120 may be associated with thegeographical region 121 of where the cluster 120 is physically located.Each cluster 120 may be located in a different geographical region 121,although in some examples, multiple clusters 120 share a samegeographical region 121.

The remote system 114 is also in communication with one or more clients10, 10 a-n via a network 112 b. The networks 112 a, 112 b may be thesame network or different networks. Each client 10 may correspond to anysuitable computing device, such as a desktop workstation, laptopworkstation, mobile device (e.g., smart phone or tablet), wearabledevice, smart appliance, smart display, or smart speaker. The clientstransmit application level requests 30, 30 a-n to the remote system 114via the network 112 b. The application level requests 30 correspond tomessages of an application protocol. For example, the application levelrequests 30 may include HTTP or HTTPS messages. That is, the applicationlevel requests 30 may correspond to HTTP(S) request messages from theclients 10. Optionally, the application level requests 30 may includethe TLS protocol to provide additional communications security.

The remote system 114, in some examples, executes a multi-cluster loadbalancer 130 that receives the application level requests 30 and aload-balancing configuration (e.g., ingress configuration) 132 thatconfigures the load balancer 130 to load balance the application levelrequests 30. Each application level request 30 includes a host name 32and a geographical location 34 associated with the source client 10. Thehost name 32 corresponds to a selection criteria (e.g., a label) thatidentifies a destination network host (i.e., one or more computers undercommon authority). For example, http://my-shop.com is a Uniform ResourceLocator (URL) that indicates the HTTP protocol and a host name ofmy-shop.com. The geographical location 34 corresponds to a physicallocation of the respective client 10 (e.g., an Internet Protocol (IP)address). Some application level requests 30 may additionally include apath name 33, e.g., the http:/my-shop.com/sports URL indicates the hostname of my-shop.com and a path name of/sports.

The load balancer 130 manages access to the clusters 120 (also referredto as destination clusters 120) that host the software application 124for a user 12. That is, using the configuration provided by theload-balancing configuration (e.g., ingress configuration) 132, the loadbalancer 130 receives application level requests 30 that are directedtoward the software application 124 on the destination clusters 120 androutes each application level request 30 to one of the destinationclusters 120 based on the geographical location 34 of the applicationlevel request 30 and the respective geographical regions 121 of thedestination clusters 120. For example, when the geographical location 34associated with a respective application level request 30 indicates thatthe application level request 30 originated from North America, the loadbalancer 130 may route the application level request 30 to the cluster120 n with a corresponding geographical region 121 n (i.e., NorthAmerica).

With continued reference to FIG. 1, in some implementations, amulti-cluster controller 200 receives the load-balancing configuration132 and, using the load-balancing configuration 132, configures amulti-cluster ingress 400. The multi-cluster ingress 400 configured bythe multi-cluster controller 200 includes a mapping of URL paths (i.e.,URL mapping 410) to software applications 124 executing on clusters 120.That is, when the multi-cluster ingress 400 receives an applicationlevel request 30 directed toward a respective software application 124executing within a respective pod 122 of a respective cluster 120, themulti-cluster ingress 400 routes the application level request 30, usingthe URL mapping 410, to the appropriate cluster 120 based on thegeographical location 34 and the associated software application 124 ofthe application level request 30. The user 12 may correspond to acreator of the destination clusters 120 for hosting the applications 124or services 123. As such, the user 12 may provide the load-balancingconfiguration 132 to the multi-cluster controller 200 of themulti-cluster load balancer 130.

Referring now to FIG. 2, the multi-cluster controller 200, in someexamples, is responsible for receiving a multi-cluster service 210 ofthe load-balancing configuration 132. For instance, the multi-clusterload balancer 130 may instantiate the multi-cluster service 210 based onthe load-balancing configuration 132. The multi-cluster service 210represents a resource that spans multiple clusters 120. In someexamples, the load-balancing configuration 132 includes a user-derivedservice name 211 (i.e., a service name derived by the user 12) thatuniquely identifies the multi-cluster service 210. The multi-clusterservice 210, in some implementations, includes a cluster selectionsection 212 that defines which clusters 120 are destination clusters 120and the load balancing properties of the destination clusters 120. Thatis, the cluster selection section 212 identifies cluster selectioncriteria 213 specified by the multi-cluster service 210 for choosingclusters 120 from a known cluster list 125 that will serve applicationlevel traffic (i.e., application level requests 30) for themulti-cluster service 210. The known cluster list 125 may include aregistry of known clusters 120, or simply refer to a cluster registrythat may be stored on the storage resources 116 of the remote system 114and include a plurality of clusters that the user 12 owns/created or hasaccess to. Using the cluster selection criteria 213, the multi-clustercontroller 200 then selects the set of destination clusters 120 from thecluster registry 125 based on each destination cluster 120 having arespective set of one or more labels 216 that satisfy the clusterselection criteria 213 specified by the multi-cluster service 210. Thatis, the selected clusters 120 may share a common set of labels 216 toenable the clusters 120 to be selected as a unit across all of theclusters 120. Optionally, the cluster selection criteria 213 specifiedby the multi-cluster service 210 includes at least one of one or moreequality-based matching requirements (e.g., environment=production) orone or more set-based matching requirements (e.g., environment in(production, qa)).

The multi-cluster service 210 may also include a service template 214that defines a service 220 that the multi-cluster controller 200instantiates/creates in each destination cluster 120 and the loadbalancer 130. In some examples, by defining the multi-cluster service210, the multi-cluster controller 200 may instantiate the derivedservices 220 in the destination clusters 120 automatically. In theexample shown, the multi-cluster controller 200 receives themulti-cluster service 210 (along with the cluster selection section 212and the service template 214) and instantiates a corresponding derivedresource (i.e., the shopping-service 220) in each destination cluster120 a, 120 b, 120 c. The multi-cluster controller 200 may automaticallymanage the entire lifecycle of the derived services 220 (e.g., creating,syncing, and deleting the services 220). The multi-cluster controller200 may instantiate and manage the derived services 220 using create,read, update, and delete (CRUD) operations. Thus, application levelrequests 30 that correspond to the multi-cluster service 210 (e.g.,shopping-service) may route via the multi-cluster ingress 400 to thederived service 220 of the appropriate destination cluster 120.

Each corresponding derived service 220 may include a unique derivedservice name 221 that is different than the derived service names 221 ofthe other derived services 220. For example, the derived service name221 has a trimmed service name portion and a unique hash portion. Thetrimmed service name portion may include the user-derived service name211 of the multi-cluster service 210 and the unique hash portion mayinclude a unique hash of the user-derived service name of themulti-cluster service 210. The respective unique derived service name221 for each derived service 220 may avoid conflicts with names ofuser-defined services 123.

In some examples, the derived services 220 create a correspondingnetwork endpoint group (NEG) 230 that includes a group of endpoints 231,231 a-n. Each endpoint 231 in the group of endpoints 231 is associatedwith a respective pod 122 of the corresponding destination cluster 120.Each endpoint 231 includes a respective internet protocol (IP) address242 and a respective port 244 for distributing application level traffic(i.e., requests 30) directly to the respective pod 122. That is, NEGs230 are resources that represent collections of IP address 242 and port244 combinations for cluster resources that operate as a backend forbackend services and each IP address 242 and port 244 combination isreferred to as a network endpoint 231. NEGs 230 may be used as backendsin backend services such as HTTP(S), Transmission Control Proxy (TCP)proxy, and SSL proxy load balancers. NEG backends facilitatedistributing traffic in a granular fashion among applications orcontainers running within pods 122 by specifying IP addresses 242 andports 244. Endpoints 231 (e.g., pods 122) in the same cluster 120 may beassigned to the NEG 230. The NEG 230 may serve as a backend for backendservices in a container load balancer 240 (i.e., a load balancer forbalancing traffic among the machines or pods 122 in the cluster 120).Each destination cluster 120 may include a corresponding NEG controller232 to program the respective NEG 230.

In other examples, the clusters 120 implement instance groups instead ofNEGs 230. Instance groups, similar to NEGs 230, group a collection ofendpoints (e.g., virtual machine instances) together as a single entityand routes requests 30 to the appropriate endpoint by using IP tables.An instance group may be a managed instance group with or without autoscaling or an unmanaged instance group.

When implementing NEGs 230 instead of instance groups, the multi-clustercontroller 200 may store names (i.e., labels) of each NEG 230 for easyretrieval by other components of the system 100. Each NEG 230 mayinclude a firewall that is managed by the NEG controller 232, allowingeach NEG to open a unique set of ports 244. Alternatively oradditionally, the multi-cluster controller 200 may instantiate afirewall controller that affects the port ranges of every destinationcluster 120. The firewall controller, for example, could ensure theentire port range is open and then allow each individual NEG controller232 to customize its respective port range.

Referring now to FIGS. 3A and 3B, in some examples, the remote system114 executes additional components to implement the container loadbalancer 240. For example, a forwarding rule 310 may direct applicationlevel requests 30 from the respective cluster's 120 global external IPaddress to an appropriate target proxy 320 (FIG. 3A). The forwardingrule 310 routes requests 30 by IP address, port, and protocol to a loadbalancing configuration consisting of the target proxy 320, a URLmapping 330 (e.g., URL mapping 410), and one or more backend services340, i.e., service(s) 123 (FIG. 1). Each forwarding rule 310 may providea single global IP address for the cluster 120. The target proxy 320terminates connections (e.g., HTTP and HTTPS connections) from clients10. The target proxy 320 checks each received request 30 against a URLmapping 330 to determine which backend service 340 is appropriate forthe request 30. When routing HTTPS connections, the target proxy 320 mayinclude one or more Secure Sockets Layer (SSL) certificate toauthenticate communications between the load balancer 240 and the client10.

As shown in FIG. 3B, unlike instance groups, which route traffic via IPtable rules to containers (e.g., pods) 122 (which may or may not be inthe same node/Virtual Machine), NEGs 230 allow for traffic to be routeddirectly to the containers (e.g., pods) 122 that should receive thetraffic (i.e., requests 30), which eliminates an extra network hop. Thereduced network hops improves both the latency and throughput of thenetwork.

The URL mapping 330 defines matching patterns for URL-based routing ofrequests 30 to the appropriate backend services 340. In some examples, adefault service 340 is defined to handle any requests 30 that do notmatch a specified host rule or path matching rule. Optionally, themulti-cluster controller 200 may create a derived default service in thedestination clusters 120. For content-based routing of requests 30, theURL mapping 330 divides requests 30 by examining URL components to sendrequests 30 to different sets of backends 340. Multiple backend services340 may be referenced from the URL mapping 330.

The backend services 340 direct incoming requests 30 to one or moreendpoints of attached NEGs 230. The backend service 340 directs eachrequest 30 to an appropriate endpoint of one of the connected NEGs 230based on, for example, serving capacity, zone, and instance health ofits attached backends. Endpoint serving capacity may be based on CPU orrequests per second (RPS) (i.e., an amount of requests 30 the endpointcan process per second). Each backend service 340 may also specify whichhealth checks to perform against the endpoints of the NEGs 230.

Referring now to FIG. 4, the multi-cluster controller 200 manages themulti-cluster ingress 400 and the multi-cluster service 210 defined bythe multi-cluster ingress 400 using the user-derived service name 211.The multi-cluster ingress 400 includes Layer 7 protocol and terminationsettings (e.g., Transport Layer Security (TLS) certificates) and the URLmapping 410 specifies a list of one or more host names 412 and/or URLpaths that map to one or more services 123 executing on the destinationclusters 120. Each destination cluster 120 includes a respective derivedservice 220 that communicates with the multi-cluster service 210. Foreach application level request 30 directed toward a software application124 (or service 123) the multi-cluster controller 200 receives, themulti-cluster controller 200 determines whether the host name 32 of thereceived application level request 30 includes one of the host names 412in the list of one or more host names 412 specified by the URL mapping410. Instead of or in addition to, the controller 200 may determinewhether the URL path 33 of the received application level request 30includes one of the paths in the list of paths 413 specified by the URLmapping 410. When the host name 32 (and/or path 33) of the receivedapplication level request 30 includes one of the host names 412 (and/orpaths 413) in the list, the multi-cluster controller 200 forwards thereceived application level request 30 to the multi-cluster service 210associated with the application 124 or service 123 (e.g.,shopping-service). Here, the multi-cluster service controller 200 istasked with load-balancing the received application level request 30 tothe respective destination service 220 of one of the destinationclusters 120, 120 a-c executing the deployed service 123. In someimplementations, the multi-cluster service controller 200 determineswhich destination cluster 120 is closest to the geographical location 34of the request 30 (e.g., the location 34 associated with the client 10that transmitted the request 30) based on the respective geographicalregions 121 a-c of the destination clusters 120. The multi-clustercontroller 200, via routing decisions defined by the multi-clusterservice 210, may route the application level request 30 to thedestination cluster 120 that has the respective geographical region 121that is closest to the geographical location 34 associated with theclient 10 of the application level request 30.

In the example shown, a client 10 a is located in Tokyo, a client 10 bis located in San Jose, and a client 10 c is located in Boston. Furthera set of destination clusters 120 executing a shopping-service 123,includes a first cluster 120 a associated with a geographical region 121a of Tokyo, a second cluster 120 b associated with a geographical region121 b of San Francisco, and a third cluster 120 c associated with ageographical region 121 c of New York City. Each client 10 a, 10 b, 10 ctransmits a respective application level request 30 a, 30 b, 30 creceived by the controller 200. The controller 200, based on thegeographical location 34 associated with the requests 30 (i.e., Tokyo,San Jose, and Boston), routes the request 30 a to the cluster 120 a, therequest 30 b to the cluster 120 b, and the request 30 c to the cluster120 c. In some examples, the multi-cluster controller 200 routes eachrequest 30 based on the cluster 120 associated with the lowest latency(i.e., an amount of time the request 30 takes to travel from the client10 to the respective cluster 120). That is, each destination cluster 120has a respective latency from the client 10, and the multi-clustercontroller 200 may route the request 30 to the cluster 120 with thelowest latency of each destination cluster 120 at any given instance intime. In other examples, the multi-cluster controller 200 routes eachrequest based on equality matching a region label that is associatedwith the geographical location 34 of the request and a region label thatis associated with the geographical region 121 of the cluster 120. Forexample, the request 30 may include a region label that corresponds to“Asia” and the multi-cluster ingress 400 may route the request 30 to acluster with a matching region label (i.e., “Asia).

In some examples, the controller 200 routes the requests 30 based onrespective load balancing (LB) attributes 420 specified by themulti-cluster service 210. For example, application level requests 30may always be routed to the closest (i.e., closest geographically)available cluster 120. In some implementations, the clusters 120 willscale automatically (e.g., increase or decrease the number of containers(e.g., pods) 122 within each cluster 120) to accommodate client demand.In this example, each cluster has, in practice, limitless resources andthus, clients 10 will always be routed to the nearest cluster 120. Byautomatically scaling the number of resources on a per-cluster basisbased on client demand, an amount of utilization (i.e., the percentageof resources in use versus the total available resources) per cluster120 remains high. In the example of FIG. 4, when the clusters 120 haveunlimited capacity to accommodate client demand, the clusters 120 maydynamically scale to meet end user demand such that when the loadbalancer 130 is receiving a greater number of application level requests30 (i.e., requests per second) from Tokyo than from San Jose and Boston,the first cluster 120 a within the geographical region 121 a of Tokyomay scale up the number of resources/containers 122 (e.g., pods) to meetthe increase in end user demand. At least one of the second and thirdclusters 120 b, 120 c within the other geographical regions 121 b, 121 cmay also scale-down based on end user demand at the correspondinggeographical locations 34. In these auto-scaling scenarios where theload balancer 130 routes requests 30 to the nearest geographical region121, the clusters 120 may be required to synchronize state amongst oneanother in order to provide stateful services 123. The load balancer 130may continuously update based on the dynamic capacity at each of theclusters 120.

In other implementations, the clusters 120 have a fixed resourcecapacity (i.e., the clusters 120 do not scale). In this situation, priorto routing the application level request 30, the multi-clustercontroller 200, for each destination cluster 120, determines whether anumber of application level requests 30 (e.g., requests per second)currently routed to the destination cluster 120 satisfies a maximumrequest rate. When the number of application level requests 30 satisfiesthe maximum request rate, the multi-cluster controller 200 preventsrouting of the application level request 30 to the destination cluster120. That is, the load balancing attributes 420 may include maximumrequest rates (i.e., maximum RPS) and in this situation, if the closestcluster based on the geographical region 121, as discussed above, hassatisfied or exceeded its threshold RPS, the multi-cluster ingress 400may route the request 30 to the next closest cluster 120 (e.g., based onthe latency or the region labels). If the second closest cluster 120also has exceeded its maximum RPS, the multi-cluster ingress 40 may moveto the third closest cluster 120, and so on. Moreover, the fixedresource capacity associated with at least one of the destinationclusters 120 may be different than the fixed resource capacitiesassociated with the other destination clusters 120.

The load balancing attributes 420 may additionally or alternativelyinclude multi-cloud and/or hybrid load balancing attributes that causeapplication level requests 30 to route to the nearest geographic cluster120 with capacity to accommodate the request 30. The cluster 120 may bein another cloud computing network or even at the same geographicallocation 34 (e.g., on-premises) that the application level request 30originated. This enables highly available services that are resilient tomultiple regional outages in a single cloud computing network and easesthe launch of a new cloud computing network.

Each cluster 120 may receive individualized load balancing attributes420 or the same attributes 420 may be applied to all destinationclusters 120. When the user 12 provides no load balancing attributes420, the multi-cluster ingress 400 may route based on default behavior(e.g., the cluster 120 with the lowest latency).

In some implementations, the load balancing attributes 420 includedata-locality routing attributes. That is, the load balancing attributesmay route application level requests 30 to a cluster 120 based onHTTP(S) header information (e.g., HTTP cookies). This allows clients 10to have their application level requests 30 routed to geographicallocations/regions 121 of clusters 120 already hosting their data andhelp meet any data residency requirements or laws. As such, only asingle IP address needs to be published for the underlying service 123executing across the set of destination clusters 120. Data residency isgenerally defined as requirements that client data must be processedand/or stored within a specific country's borders. Optionally, theclusters 120 synchronize data between each other in order to servemultiple sets of clients 10 simultaneously. Here, theresources/containers/pods 122 may scale up or down within the respectiveclusters based on end user demand. The synchronized data also allows forapplication level requests 30 to be rerouted to alternate clusters 120in the case a cluster 120 fails or is otherwise unhealthy. The loadbalancing attributes 420 include client-based routing where applicationlevel requests 30 are routed to services within a single cluster basedon the HTTP(S) header information, such as HTTP cookies or geo-headers.This allows the load balancer 130 to easily group and route clients 10to different services.

The load balancing attributes 420 may also include attributes fortraffic splitting. Traffic splitting attributes enable the load balancer130 to route application level requests 30 to the clusters 120 based ona percentage (%) split or RPS ratio among the clusters 120 defined bythe user 12. That is, each cluster may be assigned (e.g., by the user12) a percentage of total traffic (i.e., application level requests 30),and the controller 200 may randomly rout application level requests 30to the cluster 120 based on the assigned percentages. Such trafficsplitting eases the migration of workloads to a cluster 120 in a newgeographical region 121 as the cluster 120 in the new geographicalregion 121 may be brought up slowly (i.e., start with a small percentagethat increases over time, sometimes referred to as a canary deployment).Load balancing attributes 420 that specify attributes for trafficsplitting may enable multi-region splitting or intra-region splitting.In multi-region splitting, traffic may be split across geographicalregions 121. As such, multiple application level requests 30 from a sameclient 10 in a given geographical region 34 may be routed to clusters120 in more than one geographical region 121. For instance, the client10 c in Boston could issue multiple application level requests 30,whereby the load balancer 130 routes a portion of these requests 30 tothe third destination cluster 120 c in the geographical region 121 cassociated with New York City and routes a remaining portion of theserequests 30 to the first destination cluster 120 a in the geographicalregion 121 a associated with Tokyo. In intra-region splitting, trafficmay only be split within the same geographical region 121. That is,application level requests 30, with intra-region splitting, may only besplit within the same geographical region 121, while cross-regiontraffic is unaffected. For instance, clients 10 in Tokyo may be splitamong two separate clusters 120 located in a geographical region 121associated with Asia, but not be routed to a cluster with a geographicalregion 121 associated with Europe. The load balancing attributes 420 mayalso enable intra-cluster traffic splitting. With intra-cluster trafficsplitting, application level requests 30 may be randomly routed toservices within a single cluster 120 based on assigned percentages(i.e., assigned by the load balancing attributes 420). This enables thetesting of, for example, new versions of services. That is, a newversion of a service may be routed a small percentage of traffic fortesting while the majority of traffic is routed to the original versionof the service.

FIG. 5 is a flowchart of an example method 500 for load balancingapplication level requests 30 across a multi-cluster containerizedorchestration system 100. The method 500 may be described with referenceto FIGS. 1-4. The method 500 starts at operation 502 with receiving, atdata processing hardware 118, a load-balancing configuration 132 for amulti-cluster load balancer 130 managing access to a set of destinationclusters 120 hosting a software application 124 deployed by a user 12.The multi-cluster load balancer 130 is configured to use theload-balancing configuration 132 to load balance application leveltraffic 30 associated with the software application 124 across the setof destination clusters 120. Each destination cluster 120 includes atleast one container 122 executing the software application 124 and arespective geographical region 121 that is the same or different than atleast one other geographical region 121 associated with another one ofthe destination clusters 120 in the set of destination clusters.

At operation 504, the method 500 includes receiving, at the dataprocessing hardware 118, an application level request 30 directed towardthe software application 124 hosted across the set of destinationclusters 120. The application level request 30 is received from a client10 and includes a host name 32 and a geographical location 34 associatedwith the client 10. The application level request 30 can also include apath name 33. At operation 506, the method 500 includes routing, by thedata processing hardware 118, the application level request 30 to one ofthe destination clusters 120 in the set of destination clusters based onthe geographical location 34 of the application level request 30 and therespective geographical regions 121 of the set of destination clusters120.

FIG. 6 is schematic view of an example computing device 600 that may beused to implement the systems and methods described in this document.The computing device 600 is intended to represent various forms ofdigital computers, such as laptops, desktops, workstations, personaldigital assistants, servers, blade servers, mainframes, and otherappropriate computers. The components shown here, their connections andrelationships, and their functions, are meant to be exemplary only, andare not meant to limit implementations of the inventions describedand/or claimed in this document.

The computing device 600 includes a processor 610, memory 620, a storagedevice 630, a high-speed interface/controller 640 connecting to thememory 620 and high-speed expansion ports 650, and a low speedinterface/controller 660 connecting to a low speed bus 670 and a storagedevice 630. Each of the components 610, 620, 630, 640, 650, and 660, areinterconnected using various busses, and may be mounted on a commonmotherboard or in other manners as appropriate. The processor 610 canprocess instructions for execution within the computing device 600,including instructions stored in the memory 620 or on the storage device630 to display graphical information for a graphical user interface(GUI) on an external input/output device, such as display 680 coupled tohigh speed interface 640. In other implementations, multiple processorsand/or multiple buses may be used, as appropriate, along with multiplememories and types of memory. Also, multiple computing devices 600 maybe connected, with each device providing portions of the necessaryoperations (e.g., as a server bank, a group of blade servers, or amulti-processor system).

The memory 620 stores information non-transitorily within the computingdevice 600. The memory 620 may be a computer-readable medium, a volatilememory unit(s), or non-volatile memory unit(s). The non-transitorymemory 620 may be physical devices used to store programs (e.g.,sequences of instructions) or data (e.g., program state information) ona temporary or permanent basis for use by the computing device 600.Examples of non-volatile memory include, but are not limited to, flashmemory and read-only memory (ROM)/programmable read-only memory(PROM)/erasable programmable read-only memory (EPROM)/electronicallyerasable programmable read-only memory (EEPROM) (e.g., typically usedfor firmware, such as boot programs). Examples of volatile memoryinclude, but are not limited to, random access memory (RAM), dynamicrandom access memory (DRAM), static random access memory (SRAM), phasechange memory (PCM) as well as disks or tapes.

The storage device 630 is capable of providing mass storage for thecomputing device 600. In some implementations, the storage device 630 isa computer-readable medium. In various different implementations, thestorage device 630 may be a floppy disk device, a hard disk device, anoptical disk device, or a tape device, a flash memory or other similarsolid state memory device, or an array of devices, including devices ina storage area network or other configurations. In additionalimplementations, a computer program product is tangibly embodied in aninformation carrier. The computer program product contains instructionsthat, when executed, perform one or more methods, such as thosedescribed above. The information carrier is a computer- ormachine-readable medium, such as the memory 620, the storage device 630,or memory on processor 610.

The high speed controller 640 manages bandwidth-intensive operations forthe computing device 600, while the low speed controller 660 manageslower bandwidth-intensive operations. Such allocation of duties isexemplary only. In some implementations, the high-speed controller 640is coupled to the memory 620, the display 680 (e.g., through a graphicsprocessor or accelerator), and to the high-speed expansion ports 650,which may accept various expansion cards (not shown). In someimplementations, the low-speed controller 660 is coupled to the storagedevice 630 and a low-speed expansion port 690. The low-speed expansionport 690, which may include various communication ports (e.g., USB,Bluetooth, Ethernet, wireless Ethernet), may be coupled to one or moreinput/output devices, such as a keyboard, a pointing device, a scanner,or a networking device such as a switch or router, e.g., through anetwork adapter.

The computing device 600 may be implemented in a number of differentforms, as shown in the figure. For example, it may be implemented as astandard server 600 a or multiple times in a group of such servers 600a, as a laptop computer 600 b, or as part of a rack server system 600 c.

Various implementations of the systems and techniques described hereincan be realized in digital electronic and/or optical circuitry,integrated circuitry, specially designed ASICs (application specificintegrated circuits), computer hardware, firmware, software, and/orcombinations thereof. These various implementations can includeimplementation in one or more computer programs that are executableand/or interpretable on a programmable system including at least oneprogrammable processor, which may be special or general purpose, coupledto receive data and instructions from, and to transmit data andinstructions to, a storage system, at least one input device, and atleast one output device.

A software application (i.e., a software resource) may refer to computersoftware that causes a computing device to perform a task. In someexamples, a software application may be referred to as an “application,”an “app,” or a “program.” Example applications include, but are notlimited to, system diagnostic applications, system managementapplications, system maintenance applications, word processingapplications, spreadsheet applications, messaging applications, mediastreaming applications, social networking applications, and gamingapplications.

These computer programs (also known as programs, software, softwareapplications or code) include machine instructions for a programmableprocessor, and can be implemented in a high-level procedural and/orobject-oriented programming language, and/or in assembly/machinelanguage. As used herein, the terms “machine-readable medium” and“computer-readable medium” refer to any computer program product,non-transitory computer readable medium, apparatus and/or device (e.g.,magnetic discs, optical disks, memory, Programmable Logic Devices(PLDs)) used to provide machine instructions and/or data to aprogrammable processor, including a machine-readable medium thatreceives machine instructions as a machine-readable signal. The term“machine-readable signal” refers to any signal used to provide machineinstructions and/or data to a programmable processor.

The processes and logic flows described in this specification can beperformed by one or more programmable processors, also referred to asdata processing hardware, executing one or more computer programs toperform functions by operating on input data and generating output. Theprocesses and logic flows can also be performed by special purpose logiccircuitry, e.g., an FPGA (field programmable gate array) or an ASIC(application specific integrated circuit). Processors suitable for theexecution of a computer program include, by way of example, both generaland special purpose microprocessors, and any one or more processors ofany kind of digital computer. Generally, a processor will receiveinstructions and data from a read only memory or a random access memoryor both. The essential elements of a computer are a processor forperforming instructions and one or more memory devices for storinginstructions and data. Generally, a computer will also include, or beoperatively coupled to receive data from or transfer data to, or both,one or more mass storage devices for storing data, e.g., magnetic,magneto optical disks, or optical disks. However, a computer need nothave such devices. Computer readable media suitable for storing computerprogram instructions and data include all forms of non-volatile memory,media and memory devices, including by way of example semiconductormemory devices, e.g., EPROM, EEPROM, and flash memory devices; magneticdisks, e.g., internal hard disks or removable disks; magneto opticaldisks; and CD ROM and DVD-ROM disks. The processor and the memory can besupplemented by, or incorporated in, special purpose logic circuitry.

To provide for interaction with a user, one or more aspects of thedisclosure can be implemented on a computer having a display device,e.g., a CRT (cathode ray tube), LCD (liquid crystal display) monitor, ortouch screen for displaying information to the user and optionally akeyboard and a pointing device, e.g., a mouse or a trackball, by whichthe user can provide input to the computer. Other kinds of devices canbe used to provide interaction with a user as well; for example,feedback provided to the user can be any form of sensory feedback, e.g.,visual feedback, auditory feedback, or tactile feedback; and input fromthe user can be received in any form, including acoustic, speech, ortactile input. In addition, a computer can interact with a user bysending documents to and receiving documents from a device that is usedby the user; for example, by sending web pages to a web browser on auser's client device in response to requests received from the webbrowser.

A number of implementations have been described. Nevertheless, it willbe understood that various modifications may be made without departingfrom the spirit and scope of the disclosure. Accordingly, otherimplementations are within the scope of the following claims.

What is claimed is:
 1. A computer-implemented method when executed bydata processing hardware causes the data processing hardware to performoperations comprising: receiving, from one or more clients, a pluralityof application level requests directed to a software application hostedon multiple regional zones of a distributed system, each regional zoneof the multiple regional zones comprising a corresponding cluster ofcontainer pods defining to a respective node group, the correspondingcluster comprising at least one container pod executing the softwareapplication; routing one or more of the application level requests ofthe plurality of application level requests to the respective node groupof the corresponding cluster of a particular regional zone among themultiple regional zones; determining whether the at least one containerpod executing the software application of the corresponding cluster ofthe particular regional zone is able to satisfy a traffic loadassociated with the one or more application level requests routed to therespective node group of the corresponding cluster of the particularregional zone; and when the at least one container pod executing thesoftware application of the corresponding cluster of the particularregional zone is unable to satisfy the traffic load associated with theone or more application level requests routed to the respective nodegroup of the corresponding cluster of the particular regional zone,scaling the respective node group of the corresponding cluster of theparticular regional zone by instantiating one or more new container podsto support the traffic load associated with the one or more applicationlevel requests.
 2. The method of claim 1, wherein routing the one ormore of the application level requests of the plurality of applicationlevel requests to the respective node group of the corresponding clusterof the particular regional zone comprises load balancing the pluralityof application level requests across the multiple regional zones.
 3. Themethod of claim 1, wherein routing the one or more of the applicationlevel requests of the plurality of application level requests to therespective node group of the corresponding cluster of the particularregional zone comprises load balancing the plurality of applicationlevel requests across the multiple regional zones based on a loadbalancing configuration, the load balancing configuration routing theplurality of application level requests to the closest geographicallyavailable cluster of container pods.
 4. The method of claim 1, whereinrouting the one or more of the application level requests of theplurality of application level requests to the respective node group ofthe corresponding cluster of the particular regional zone comprises loadbalancing the plurality of application level requests across themultiple regional zones based on a load balancing configuration, theload balancing configuration established by a respective client of theone or more clients.
 5. The method of claim 1, wherein the particularregional zone is based on a geographical location of the one or moreapplication level requests.
 6. The method of claim 1, wherein eachrespective node group is centrally managed by a multi-cluster service.7. The method of claim 1, wherein instantiating one or more newcontainer pods to support the traffic load associated with the one ormore application level requests routed to the respective node group ofthe corresponding cluster of the particular regional zone comprisesgenerating the one or more new container pods using a launch template,the launch template configured to create the software application in arespective new container pod.
 8. The method of claim 1, wherein therespective node group comprises a respective internet protocol (IP)address and a respective port for distributing application level trafficdirectly to one or more container pods.
 9. The method of claim 1,wherein each application level request of the plurality of applicationlevel requests comprises HyperText Transfer Protocol (HTTP).
 10. Themethod of claim 1, wherein each application level request of theplurality of application level requests comprises HyperText TransferProtocol Secure (HTTPS).
 11. The method of claim 1, wherein at least aportion of each application level request of the plurality ofapplication level request comprises a transport layer security (TLS)protocol.
 12. A system comprising: data processing hardware; and memoryhardware in communication with the data processing hardware, the memoryhardware storing instructions that when executed on the data processinghardware cause the data processing hardware to perform operationscomprising: receiving, from one or more clients, a plurality ofapplication level requests directed to a software application hosted onmultiple regional zones of a distributed system, each regional zone ofthe multiple regional zones comprising a corresponding cluster ofcontainer pods defining to a respective node group, the correspondingcluster comprising at least one container pod executing the softwareapplication; routing one or more of the application level requests ofthe plurality of application level requests to the respective node groupof the corresponding cluster of a particular regional zone among themultiple regional zones; determining whether the at least one containerpod executing the software application of the corresponding cluster ofthe particular regional zone is able to satisfy a traffic loadassociated with the one or more application level requests routed to therespective node group of the corresponding cluster of the particularregional zone; and when the at least one container pod executing thesoftware application of the corresponding cluster of the particularregional zone is unable to satisfy the traffic load associated with theone or more application level requests routed to the respective nodegroup of the corresponding cluster of the particular regional zone,scaling the respective node group of the corresponding cluster of theparticular regional zone by instantiating one or more new container podsto support the traffic load associated with the one or more applicationlevel requests.
 13. The system of claim 12, wherein routing the one ormore of the application level requests of the plurality of applicationlevel requests to the respective node group of the corresponding clusterof the particular regional zone comprises load balancing the pluralityof application level requests across the multiple regional zones. 14.The system of claim 12, wherein routing the one or more of theapplication level requests of the plurality of application levelrequests to the respective node group of the corresponding cluster ofthe particular regional zone comprises load balancing the plurality ofapplication level requests across the multiple regional zones based on aload balancing configuration, the load balancing configuration routingthe plurality of application level requests to the closestgeographically available cluster of container pods.
 15. The system ofclaim 12, wherein routing the one or more of the application levelrequests of the plurality of application level requests to therespective node group of the corresponding cluster of the particularregional zone comprises load balancing the plurality of applicationlevel requests across the multiple regional zones based on a loadbalancing configuration, the load balancing configuration established bya respective client of the one or more clients.
 16. The system of claim12, wherein the particular regional zone is based on a geographicallocation of the one or more application level request.
 17. The system ofclaim 12, wherein each respective node group is centrally managed by amulti-cluster service.
 18. The system of claim 12, wherein instantiatingone or more new container pods to support the traffic load associatedwith the one or more application level requests routed to the respectivenode group of the corresponding cluster of the particular regional zonecomprises generating the one or more new container pods using a launchtemplate, the launch template configured to create the softwareapplication in a respective new container pod.
 19. The system of claim12, wherein the respective node group comprises a respective internetprotocol (IP) address and a respective port for distributing applicationlevel traffic directly to one or more container pods.
 20. The system ofclaim 12, wherein each application level request of the plurality ofapplication level requests comprises HyperText Transfer Protocol (HTTP).21. The system of claim 12, wherein each application level request ofthe plurality of application level requests comprises HyperText TransferProtocol Secure (HTTPS).
 22. The system of claim 12, wherein at least aportion of each application level request of the plurality ofapplication level request comprises a transport layer security (TLS)protocol.